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Abstract 

We study the following critically coupled perturbed Brezis-Nirenberg problem 
(Bose-Einstein condensation system for the higher dimensional case): 

u > 0, w > in ri, u = w = on dVL. 

Here, U C M^(iV > 5) is a smooth bounded domain, 2* 
— Ai(ri) < Ai,A2 < 0, ^i,A*2 > and /3 7^ 0, where Ai(17) is the first 
eigenvalue of —A with the Dirichlet boundary condition. When /? = 0, this 
turn to be the well-known Brezis-Nirenberg problem. It is interesting that 
we can show that this system has a positive least energy solution for any 
(3 . We also study the limit behavior of the least energy solutions in 
case l3 — > —00 and phase separation is expected. As an application of the 
phase separation, we obtain a least energy sign-changing solution of the 
Brezis-Nirenberg problem in case > 6. In case Ai = A2, the uniqueness of 
the positive least energy solutions is also studied. It turns out that some quite 
different phenomenons happen comparing to the special case iV = 4, which 
has been studied by the authors in another paper. 
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1 Introduction 



In this paper we consider the fohowing coupled nonhnear Schrodinger equations: 

{-Au + Xiu = niu'^P-^ + l3uP-^vP, xen, 
-Av + X2V ^ H2V^P-^ + (3vP'^uP, xen, (1.1) 
u > 0, V > in £7, u = u = on dfl. 

where ft — M.^ or fl C is a smooth bounded domain, p > 1 and p < 2* /2 if 
N > 3, fii, ^2 > and /3 7^ is a couphng constant. In the case p = 2, the cubic 
system (II. ip appears in many physical problems, especially in nonlinear optics 
and Bose-Einstein condensation. We refer for this to [3J [501 IHl HI] j which also 
contain information about the physical relevance of non-cubic nonlinearities. 
In the subcritical case p < 2*/2, the existence of solutions have received great 
interest recently, see [31 [71 [THl [^Sl [5^1 HHl IMl 133 and references therein. 

Remark that, all the papers mentioned above deal with the subcritical case. 
To the best of our knowledge, there is no existence results for (jl.ip in the 
critical case 2p — 2* in the literature. Recently, the authors considered the 
special critical case p = 2 and A'^ = 4 in [T7] . 

In this paper, we consider the existence of least energy solutions of (|l.ip in 
more general critical case with higher dimensions. In the sequel we assume that 

N>5 and 2p = 2*. (1.2) 

It turns out that different phenomenons happen comparing to the case iV = 4 
([TT]), see Remarks [HH [Tl [O] and [TH below. If O = and {u,v) is any 
a solution of (|l.ip . then by the Pohozaev Identity and E' {u,v){u,v) ~ 0, it is 
easy to get that J^j^ Aiu^ + \2v'^ dx — 0, so {u,v) = (0,0) if A1A2 > 0. Here 
E is the related functional defined in ()1.4p . Therefore, in the sequel we assume 
that C is a smooth bounded domain . We call a solution (m, v) nontrivial 
if both u ^ and u ^ 0, a solution (u,f) semi-trivial if {u,v) is type of (m,0) 
or (0, v). 

Let Ai(ri) be the first eigenvalue of —A with the Dirichlet boundary condi- 
tion. Note that (jl.ip has semi-trivial solutions (u^j^,0) and (0,1*^3), where 
is a positive least energy solution of the well-known Brezis-Nirenberg problem 

- Au + XiU = p.i\u\^'~'^u, u e i?o(f7), (1.3) 

if — Ai(r^) < Ai, A2 < (see [9J). Hence, system (|l.ip is a critically coupled per- 
turbed problem of the Brezis-Nirenberg problem. The Brezis-Nirenberg problem 
(|1.3p has been studied by many authors, and it seems almost impossible to give 
a complete list of references. We refer the readers to [131 HH HH HH [33] and 
references therein. 

We are concerned with nontrivial solutions of (jl.ip . Define H :— -ffo(il) x 
Hq (fi) . It is well known that solutions of (jl.ip correspond to the critical points 
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of functional : H — > R given by 



2p 



{fiM''' + w\v\p+fi2\v\^n- (1-4) 



We say a solution {u, v) of (jl.ip is a least energy solution, if {u, v) is nontrivial 
and E{u,v) < E{(p,ip) for any other nontrivial solution (<p,?/') of ()l.ip . As in 
[25], we define 



M = {{u,v)eH,u^O,v^O, / (|Vup + Aiu2)= / (^i|upP + /3|Mnt^r), 

Jn Jn 



Then any nontrivial solutions of ()l.ip has to belong to M. Take (p, ?A € C(5"(il) 
with (p,ip ^ and supp{ip) D supp{ip) = 0, then there exists ^1,^2 > such that 
{tiip,t2ip) &M. SoM ^0. We set 

B:= inf £;(u,w)= inf 4 / (I^^P + ^i^^^ + I^^P + ^2"^) da;. (1.5) 

{u,v)GM (u.,v)eM N Jq 

First we consider the case — Ai(57) < Ai = A2 = A < 0. By [9] the Brezis- 
Nirenberg problem 

-Au + Xu^ \uf^^u, u e Hl{n) (1.6) 
has a positive least energy solution lu with energy 

Moreover, 

[ {\Vu\^ + \u^)dx>{NBif'^ ( [ \uf d:^' , V u G ^[[(l}). (1.8) 
Consider 

Vifcp-i + pkP/^'Hp/^ = 1, 

/3fcP/2K2-i + /zaZ^'-i = 1, (1.9) 
fc > 0, l>0. 

We will prove in Lemma [2A] that there exists (fco, '0), such that 

{koJo) satisfies (|1.9p and fco — min{fc : (fc, /) is a solution of (|1.9p }. (1-10) 
Our first result deals with the special case Ai = A2. 
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Theorem 1.1. Assume that — Ai(r2) < Ai = A2 = A < 0. Let {ko,lo) in HI. 10]) . 
Then for any /3 > 0, (^/kouj, \/Iquj) is a positive solution of U.l]) . 

Moreover, if P > j^^max{iii, 112}, then E {y/kou! , ^/Iqoj) — B, that is, 
{y/koUJ, \/loUj) is a positive least energy solution of 

Remark 1.1. (1) In the case iV = 4 and 2p = 2*, Theorem 1.1] said that 
il.l\) has no nontrivial nonnegative solution if 13 £ [inin{/ii, /i2}, niax{/ii, /i2}] 
and fii 7^ /i2 • Therefore, the general case N > 5 is quite different from the case 
N = A. As we will see in Section 2, the proof of Theorem \1.1\ is much more 
delicate than the proof of \17[ Theorem 1.1]. 

(2) Similarly as in |_/7| Remark 1.1], we can prove that, if Q is starshaped, 
the assumption — Ai(f2) < A < m Theorem ] 1.1\ is optimal. 

In case Ai = A2, we also study the uniqueness of the least energy solutions. 

Theorem 1.2. Assume that — Ai(il) < Ai = A2 = A < and let {ko,lo) in 
m.lO]} . There exists /?o > ^^^^ max{/ii, ^2} determined by (/ii,/i2), and assume 
that l3 > I3q. Let {u,v) be any a positive least energy solution of il.l]) . then 
{u,v) ~ {\/k^U, \/TqU), where U is a positive least energy solution of \1.6\) . In 
particular, the positive least energy solution of lil.l]) is unique if CZ is a 
ball. 

Remark 1.2. (1) We can give a precise definition of /3q (see J/^.^p in Section 
4). In particular, if = ^2, then /Sq = ^^^^ max{^i, ^12}- 

(2) For the case p = 2 and N < 3, some uniqueness results about system 
lll.l]) were introduced in \37}j . However, their proofs heavily depend on p — 2, 
and can not work here. 

Now, let us consider the general case — Ai(ri) < Ai,A2 < 0. Without loss 
of generality, we may assume that Ai < A2 . Our following result also deal with 
the case (3 < 0. 

Theorem 1.3. Assume that —Xi{il) < Ai < A2 < 0. Then system U.l]) has a 
positive least energy solution {u,v) with E{u,v) = B for any /3 7^ 0. 

Remark 1.3. For the general case Ai < A2, when N = A and 2p = 2* , \1'7\ 
Theorem 1.3] said that il.l]) has a positive least energy solution for any 

/3C (-cx),0)U(0,/3i)U(/32,+oo), 

where Pi,i = 1,2 are some positive constants satisfying 

/3i < mm{fii, fi2} < ma.x{fii, fi2} < P2- 

That is, we do not know whether the least energy solution exists or not if 13 £ 
, ^2] • Comparing this with Theorem \1.!A we see that the general case N > ^ 
is completely different from the special case N = 4. 
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Now, we study the limit behavior of the positive least energy solutions in 
the repulsive case /3 — >■ — cxj. It is expected that components of the limiting 
profile tend to separate in different regions of the underlying domain Q. This 
phenomenon, called phase separation, has been well studied for L°° -bounded 
positive solutions of system in subcritical case 2p < 2* by [351 1311 HH]- The 
critical case = 4 and p = 2 was studied by [T7]. For other kinds of elliptic 
systems with strong competition, phase separation has also been well studied, 
we refer to [TTl jT^l [H] and references therein. Denote {u > 0} := {x G fl : 
u{x) > 0}. Then we have the following result. 

Theorem 1.4. Assume that — Ai(0) < Ai < A2 < 0. Let /3„ < 0, n e N satisfy 
Pn — —00 as n 00, and w„) he the positive least energy solutions of 
with j5 = Pn which exists by Theorem \l.S\ Then /3„uPw^ dx ^ as n ^ 00, 
and passing to a subsequence, one of the following conclusions holds. 

(1) M„ — >■ Woo strongly in H^{^) and Vn weakly in H^(VL) (so w„ -t> for 
almost every a; G j, where Uoo is a positive least energy solution of 

-An + \iu = u e HQ{n). 

(2) Vn Voo strongly in HqIJI) and u„ ^ weakly in HQ{fl) (so Un ^ for 
almost every x d fl), where Voo is a positive least energy solution of 

-Av + X2V = ^i2\vf^^v, V e H^{n). 

(3) (m„,w„) — >■ (uoo,Woo) strongly in H}^{^) x i?o(ri) and Urx, ■ Voo = 0, where 
Uoo G C{Q) is a positive least energy solution of 

-An + Xiu = ^i\uf'^^u, u e HI{{uoo > 0}), 

and Voo G C{Q) is a positive least energy solution of 

-Av + X2V = fl2\vf^^V, V e H^{{Vao > 0}). 

Furthermore, both {voo > 0} and {woo > 0} are connected domains, and 
{Voo > 0} = n\{uoo > 0}. 

In particular, if N >6, then only (3) holds. 

Remark 1.4. Theorem \1.4\ has been proved for the case N — A and 2p — 2* by 
the authors (11 7^1 ). The novelty here is that we can exclude conclusions (l)-(2) 
if N > 6, which can not be proved for the case N = A. In the proof of Theorem 
\1.4[ (3), a key point is to prove the continuity of u^o o,nd Voo- We remark here 
that, our proof of the continuity of u^o and f 00 is completely different from that 
in \17l for the case N — A, and can be used to the case N — A. 

As an application of Theorem II. 41 we turn to consider the existence of least 
energy sign-changing solutions of the Brezis-Nirenberg problem 
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(1.11) 



where — Ai(r2) < Ai < 0. Define 

J(w) = i [ {\\/u\^ + \iu^)dx- [ fiilu]^" dx, ueH^ifl), 
2 Jn 2* 

So := {u e i?o'(^^)\{0} : J'iu)u = 0}, Ssc {u G H^ifl) : u+,u- e 5o}, 

Csc ■■= inf J(u), 

where :— niax{±u, 0}. We say a sign-changmg solution u of p. lip satisfying 
J{u) — Csc a least energy sign-changing solution. For some subcritical elliptic 
problems, the existence of least energy sign-changing solutions has been well 
studied, see [TOl [13 132] and references therein. For the Brezis-Nirenberg prob- 
lem (|l.lip . Cerami, Solimini and Struwe [13] showed that for > 6, problem 
(|l.lll) has at least two nontrivial solutions, one of which changes sign. Recently 
Schechter and Zou [33] showed that (jl.lip has infinitely many sign-changing 
solutions if > 7. However, to the best of our acknowledge, there seems no ex- 
istence results of the least energy sign-changing solutions in the literature. The 
main difficulty is to show the existence of some u G Ssc such that J{u) = Csc- 
Here we can answer this question in case N > 6 with the help of Theorem 11.41 
Let A'^ > 6, A2 = Ai and fi2 = Mi in Theorem 11.41 Then (moo,Woo) in Theorem 
11.41 satisfies (3), and so Uoo — Voo G Ssc- We have the following result. 

Theorem 1.5. Assume N > 6. Let {uoo,Voo) be in Theorem \1.4\ in the case 
where A2 = Ai and fJ.2 — fJ-i- Then u^o — Voo is a least energy sign-changing 
solution of hl.ll]) . 

Remark 1.5. As we will see in Section 5, Theorem \1.5\ is a direct corollary of 
the existence of a least energy sign-changing solution of the following problem 

- Au + \iu+ - \2U- = ^Li{u+f'-^ - H2{u^f'^^, ueH^ifl), (1.12) 

where — max{±M, 0}. Note that this general result has not been proved in 
the literature. Unfortunately, we can not show Theorem \1.5\ for N — 4,5. The 
reason is that we can not .show that only (3) in Theorem \1.4\ holds for N = A,5, 
which remains as a very interesting open problem. 

Since the nonlinearity and the coupling term are both critical in (jl.ip . the 
existence of nontrivial solutions of (jl.ip depends heavily on the existence of the 
least energy solution of the following limit problem 

-Au = ^i|u|2p-2y + /3\u\P-^u\v\P, X G R^, 
-Av ^ ^l2\v\^P~^v + l3\v\P~^v\u\P, xem^, (1.13) 
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where D^'^{M.^) := {u E L^{R^) : \Vu\ G L^{R^)} with norm \\u\\di.2 
ilwN |Vw|2dx)i/2. Let S be the sharp constant of D^'^(R^) ^ L^'iR^) 



\'u\^dx>s( / \uf dx]' . (1.14) 



For e > and y g R^, we consider the Aubin-Talenti instanton [1] [33] 
U^^y e D^-^{R^) defined by 



U,Jx):^[N{N~2)]^ -\ . (1.15) 

Then Ue,y satisfies — Au = in and 

/ \VU,J^dx^ f \Ue.yf dx ^ S^/\ (1.16) 

Furthermore, {U^^y : e > 0,?/ £ R^} contains all positive solutions of the 
equation —Au — |up in M^. 

jV-2 N — 2 

Note that (|1.13p has semi-trivial solutions (^^ * C/^.y, 0) and (0, ^2 " t^e.y)- 
Here, we are only interested in nontrivial solutions of (|1.13p . Define D := 
Di'2(R^) X Di'2(R^) and a hmctional / : D -> R given by 

(1.17) 

As in [25], we consider the set 

' (MiH^f + /3|unz;|P), 



{m\v\'''+f3\unv\n}- 



Then any nontrivial solutions of (|1.13p has to belong to Af. Similarly as , we 
have TV 7^ 0. We set 



A:= inf I{u,v) = inf ^/ |Vu|^ + |Vu|^ dx. 



(1.18) 



Then we have the following theorem, which plays an important role in the proof 
of Theorem 11.31 



Theorem 1.6. (1) If (3 <0, then A is not attained. 

(2) If /3 > 0, then I11.13\) has a positive least energy solution {U,V) with 
I{U, V) — A, which is radially symmetric decreasing. Moreover, 
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(2-1) if P > max{^i, ^2}, then I{y/k^Ue,y, \/loUe,y) = A, where (fco,/o) 
in U.lOp . That is, {\/koUe,y, \/hiUe,y) is a positive least energy solu- 
tion of (EH). 

(2-2) there exists < /?i < -^^2-^ niax{/ii, /i2}7 and for any < /? < f3i, 
there exists a solution {k{j3),l{/3)) of il.9\) . such that 

I{^k{P)U,^y, ^/mUe.y) >A = I{U, V). 

That is, {y/k{l3)Ui;,y, yJT{fi)Ue.y) is a different positive solution of 
ITJ3) with respect to {U,V). 

Remark 1.6. In the case N ^ 4 and 2p ^ 2* , ll7\ Theorem 1.5] said that il.l3\) 
has no nontrivial nonnegative solution if /3 € [niin{/ii, /i2}, niax{/ii, /i2}] and 
Hi ^ fj,2; {\/k(fi)U^,y, ^/l{p)Ui;^y) is a positive least energy solution of lil.lSl) if 
< /3 < min{/ii, /i2}. Therefore, the general case N > 5 is completely different 
from the case N = 4. As we will see in Section 2, the idea of proving (2) of 
Theorem \1.6\ in case < /3 < -^^2-^ niax{/ii, /i2}; which also works for the case 
j3 > j^z2 ma-xj/ii, M2}> "is completely different from that in case N — A ( llTp . 

We can also study the uniqueness of the positive least energy solutions of 
(fri3l) just as Theorem O 

Theorem 1.7. Let /3o be in Theorem \1.2\ and assume that /3 > /So- Let (a, v) be 
any a positive least energy solution of il.l3\) . Then (u, v) ~ {^/koUe^y, Vk)U^^y) 
for some s > and y G R^. 

The rest of this paper proves these theorems, and some ideas of the proofs 
are similar to those in [T7]. However, as has pointed out in Remarks II. 111. 3[ 11.41 
and ll.61 the general case TV > 5 is different from = 4, and some new ideas are 
needed. We give some notations here. Throughout this paper, we denote the 
norm of L'^{il) by \u\g — {J^ |m|'^c?x)9, the norm of i7Q(i7) by ||u|| = |Vu|2 and 
positive constants (possibly different) by C. The paper is organized as follows. 
Theorems ll.ll and ll.6l are proved in Section 2, and we will see that these proofs 
are more delicate than those in case iV = 4 (171). In Section 3, we use Nehari 
manifold approach and Ekeland variational principle to prove Theorem 11.31 for 
the case /3 < 0, and use mountain pass argument to Theorem 11.31 for the case 
/3 > 0. In Section 4, we use an elementary approach to prove Theorems 11.21 and 
11.71 Finally, we use energy estimate methods to prove Theorems 11.41 and 11.51 in 
Section 5, where some different ideas are needed. 

After the completion of this paper, we learned that (11.11) has been also 
studied in a recent preprint (23i , where the author showed the existence of least 
energy solutions for /3 > /3o > 0, where /3o is an unknown constant. For |/3| 
sufficiently small, the existence of nontrivial solutions was studied in |23| via a 
perturbation method, but the solutions obtained there seem not necessary to 
be least energy solutions. Note that Theorem 11.31 in this paper is much more 
general that those results in [33] . We also remark that any other results in our 
paper can not be found in [23] . 
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2 Proof of Theorems 11.11 and 11.61 

Define 

ai{k, I) := nikP^^ + l3k^~^l^ - 1, k>0, l>0; (2.1) 
a2{k, I) := ^12^-^ + Pl^-^k^ - 1, ? > 0, fc > 0; (2.2) 



/ii(fc) :=/3-2/p('fci-P/2_^^p/2j ^ 0<fc<Mi''-^ (2.3) 

, / , , s 2/p !_ 

h2{l) := r^/P [l^-P/^ - ^i2l''^^) , 0</<Ai2""'- (2.4) 

Then we have ai{k, hi{k)) = and a2{h2{l),l) = 0. 

Lemma 2.1. Assume that /3 > 0, f/ien equation 

ai{k,l) = 0, a2{k,l) = 0, kj>0 (2.5) 

/las a solution {ko,lo), which satisfies 

a2{k, hi{k)) < 0, V < fc < /co, (2.6) 

that is, {ko^lo) satisfies il.lU\) . Similarly, \2.5\) has a solution (fci,Zi) that sat- 
isfies 

ai(/i2(0:0 < 0, VO</<Zi. (2.7) 
Proof. Equation ai{k, I) — 0, k,l > imply that 

l^hi{k), 0<fc</i7'^. 

While, a2{k,l) = implies that ^21''^'^ + f3kP/'^ = l^-P/'^. Therefore, we turn to 
prove that 

have a solution. Note that (|2.8p is equivalent to 

Recall that N > 5 and 2p = 2*, we have 2 ~ p > and so 

lim /(fc) = +(^, /(/i^^) = < 0. 

fe-)-0+ ^1 

_ 1 

Therefore, there exists ko G (0,^]^ ''"^) such that /(fco) — and /(fc) > for 
k e (0,fco). Let Iq = hi{ko), then (fcoj^o) is a solution of (|2.5p . Moreover, (12.61) 
follows directly from /(fc) > for fc e (0, fco)- The existence of (fci,/i) that 
satisfy p.Sp and p.7p is similar. □ 
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Lemma 2.2. Assume that P > {p — 1) niax{/ii, /^2}; then hi{k) + k is strictly 

1 1 

increasing for k G ''^^] and h2{l) + l is strictly increasing for I G [0, ''^^]- 

Proof. Since for fc > 

we see that h[{k) > for < ^iifc''"^ < ^ or //ifc^^i = 1, and h[{k) < for 
< ^ikP~^ < 1. By direct computations, we deduce from h'({k) — 0, < 
HikP'^ < 1 that k = (^ip)^p^. Since /? > (p — 1) max{/xi, we have 

min ^ h[{k) = = -/3-2/PAii/''(p- > -1, 

and h[{k) > — 1 for < A: < /ij^ ''"^ with /c 7^ This imphes that 

_ 1 

hi{k) + A: is strictly increasing for k G ''^^]- Similarly, h2{l) + I is strictly 

increasing for I G [0,/i2 ''"^]- □ 

Lemma 2.3. Assume that P > {p — 1) max{/xi, /X2}. Lef {ko,lo) be in Lemma 
[E7\ Then ma.x{fii{ko + lo)P'^ , H2{ko + lo)P-^} < 1 and 

a2ik,hi{k)) <0, y < k < ko; ai{h2il),l) < 0, ^ < I < Iq. (2.10) 

Proof. By Lemma [2?2l we have 

^Ih^' > h,{ko) + ko ^ ko + lo, 

that is, ^i(fco + loY^^ < 1. Similarly, ^2(^0 + ^0)^^^ < 1- By Lemma \TT[ to 
prove (|2.10p . it suffices to prove that (fco,Zo) — {ki,li)- By (|2.6l) - (|2.7p we see 
that ki > fco,^o ^ ^1- If ki > fco, then fci + hi(ki) > kg + /ii(fco), that is, 
h2{h) + h = ki + li > ko + lo — /i2('o) + ^0, and so li > Iq, a contradiction. 
Therefore, ki = ko and Iq = h- This completes the proof. □ 

Lemma 2.4. Assume that /3 > (p — 1) max{/ii, /i2}. T/ien 

fc + ? < fco + /o, 

ai(fc,0>0, a2(fc,0>0, (2.11) 
k,l>0, (kj) ^ (0,0) 

has a unique solution (fco,^o)- 

Proof. Note that {ko, k) satisfies (PTT|) . Let (fc, /) be any a solution of (I2.1ip . 
Without loss of generality, we assume that fc > 0. If / = 0, then by fc < fco + 
and Q!i(fc,0) > we get that 

l</^ifc^-'<Mi(fco + ^o)^"\ 
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which contradicts with Lemma [2.31 Therefore / > 0. 

Assume by contradiction that k < k^. Similarly to the proof of Lemma [2?2l 

by (|2.4p it is easy to see that h2{l) is strictly increasing for < fJ,2l^^^ < ^— ^, and 

_ 1 

strictly decreasing for < ^2^*^^^ < 1- Moreover, /i2(0) = /i2(a*2 ''"^) ~ 0. 

_ 1 

Since < fc < fco = /i2(^o), there exists < ^2 < ^3 < Ai2 "^^ such that 
^12(^2) = ^^2(^3) = k and 

a2{k,l)<0 ^ h2{l)>k ^ l2<l<h. (2.12) 

Since a2{k, I) > 0, we have I < I2 or I > I3. Since ai{k, I) > 0, we have / > hi{k). 
By Lemma [2.31 we have Q;2(fc, /ii(fc)) < 0, and so I2 < hi{k) < I3. These imply 
that 

l>h- (2.13) 
On the other hand, since :— ko + lo — k > Iq, we have 

h2ih) + fco + ^0 - fc = ^12(^1) + ^1 > ^-2(^0) + I0 = ko + lo, 

that is, /i2(^i) > k. By (I2.12p we have h < h < h- By k + I < ko + Iq we have 

l<h< h, 

which contradicts with (|2.13p . Therefore, k > ko. By a similar argument, we 
also have I > Iq. Therefore, (fc, Z) = {ko,lo). This completes the proof. □ 

Proof of Theorem II. li Assume that -Ai(il) < Ai = A2 = A < 0. By 

the Sobolev inequality (|1.14l) it is standard to see that B > 0. Since /3 > 0, 
by Lemma [2.11 equation (|1.9I) has a solution (fco,'o)- Recall (|1.7p . we see that 
(a/AJocj, \/Touj) is a nontrivial solution of ()l.ip and 

< B < E{^/h>LJ,y/kLJ) = {ko + lo)Bi. (2.14) 

Now we assume that P > {p — 1) max{/ii, /i2}, and we shall prove that 
B — E{^/kQUj, a/Iqw). Let {(itn, ^n)} C be a minimizing sequence for B, that 
is, E{un,Vn) — >■ -B. Define 




By (fTS]) we have 

(7Vi?i)2/^c„ < / (|V7.„p + Aii2)= /■ (/.i|ii„|2f + /3|u„nz;„n 
Jn Jn 

<Mi< + /3</2dW2^ (2.15) 

(7VBi)2/^d„ < / (|V«„|2 + Ai;2)= /■ (M2|«j''' + /3|^i„nt'„n 

<^i2dl^pc?J^<PJ\ (2.16) 
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Since = ^ /^(|Vu„|2 + Am^ + |Vw„|2 + Xvl), by (EH]) we have 

iNB,f/^{cn+d,,) < NE{u,,,Vr,) < N{k„ + lo)B,+oil), (2.17) 
Micr^ + /3</2-idf/2 > (iVBi)2/^, (2.18) 
M2dr' + /3c5;/'</'-' > (NB,)'/^. (2.19) 

First, this means Cn,dn are uniformly bounded. Passing to a subsequence, 
we assume that c„ c and d„ d. Then by ^.15^ - ^.16^ we have fucP + 
2ljcP/'^(F/'^+li2(F >NB>0. Hence, without loss of generality, we assume that 
c> 0. If d = 0, then (^17)) implies c < (NBi)'^-^/^ {ko + lo)- By (1^1^ and 
Lemma [2.31 we get 



(iVi3i)2/^ < ^ic^'-i < Mi(fco + lof-HNB.f/'' < (NB.f/^, 

a contradiction. Therefore, c > and d > 0. Let k — (jv_Bi)^i-^/" ^'^"^ ^ ~ 
(jVBi)'i"2/» ' ^^'^^ (I^T7)) - (I^TTO)) we see that {k,l) satisfies (I^TTj) . By Lemma 
[13] we see that (A:,/) = (kg Jo). It follows that c„ A:o(iVBi)i"2/w _^ 
;o(iVBi)i-2/A' as n -> +oo, and 

7VS= lim NE{un,Vn)> lim (iVBi)2/^(c„ + d„) = 7V(A:o + Zo)-Bi- 
Combining this with (I2.14p . one has that 

B = {ko + h)Bi = E{y^oj, y%uj), 
and so {^/kout, ^/Iqoj) is a positive least energy solution of p.ip . □ 

Now we turn to the proof of Theorem 11.61 By the Sobolev inequality (|1.14p 
it is standard to see that 

A= inf -[ (|Vwp + iVvp) dx > 0. (2.20) 

Lemma 2.5. If A (resp. B) is attained by a couple {u,v) G Af (resp. {u,v) € 
AA), then this couple is a critical point of I (resp. E), provided — oo < /3 < 0. 

Proof. Let /3 < 0. Assume that [u, v) € N such that A = I{u, v). Define 

Gi{u,v)^ f \vu\'-f {ti,\u\''+m''\vn. 

G2{u,v)= f |V«p-/ (^2|H'^ + /3H^kn- 



Then there exists two Lagrange multipliers Li, L2 G K such that 
I'{u, v) + LiG[{u, v) + L2G^(m, v) = 0. 



12 



This implies < r{u,v) + LiG[{u,v) + L2G'2{u,v), (u,0) >= 0, that is, 

(2p-2)/ ^,\u\^P-(2-p) [ I3\u\p\v\Al,+L2p[ 13\u\p\v\p ^ 0. 
Similarly, < I'{u,v) + LiG[{u,v) + L2G'2{u,v), (0,w) >= implies that 

(2p-2)/ ^^\v\^P - {2 - p) [ P\u\p\v\Al2 + L,p [ l3\u\P\v\P = 0. 
Since /3 < 0, we see from Gi{u, v) ~ G2{u, u) = that 



and so 

x((2p-2)/ ^,2\v\^P-{2-p) I p\u\P\v\A > ip I p\unvf 

\ JR" JR" / \ JR" y 

From this we deduce that ii = L2 = and so I'{u,v) = 0. Similarly, if 
(u, ?;) G such that E{u, v) = S, then E'{u, v) = 0. □ 

Proof of (1) in Theorem 11.61 This proof is similar to the proof of [T71 
Theorem 1.5- (1)] in case iV = 4, but the details are more delicate. By (|1.15p 

we see that uj^. * J7i,o satisfies equation — Au = ^^u in M^. Let 

ei = (1,0, •• • ,0) e and ' 

{uii{x), vr{x)) = (wpi (a;), (^^ + ^ei)). 

Then vr^O weakly in D^''^{R^) and so ^ weakly in L'^{R'^) as R ^ +00. 
That is, 

lim / WpWp = 0. 

Note that /3 < 0. Then for i? > sufficiently large, by a similar argument 
as that of Lemma [2.11 (or see the argument of existing (i^, s^) in the proof of 
Lemma below), we see that 

{t'^ /rjv I Vuflp dx = i^/ii dx = t^P^i /jjjv dx + tPs'Pp j^^, u^rVr dx, 

/rn IVwflp = s'^^2 /rn t'if c'a:^ = s^p^2 /rn I'^f ^2: + tPsPp m^w^ dx, 

have a solution (i/j, s/j) with tR> 1 and sj^ > 1. Denote 

Di := ^1 / dx — fii / cj^^ > 0, 

JR« JR« 
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D2 / v'r dx = fj,2 / dx > 0, 

Ffl := \I3\ / u^w^ ^ 0, as i? ^ +00. 

Then 

- tl^D, ~ t^j^slFn, slD2 = s'^D2 - t^^F^,. (2.21) 
Assume that, up to a subsequence, Ir — >■ +00 as i? 00, then by 

we also have s^, — > +00. Note that 2 — p < p, we have 



i^L*! - 4 ^-Di > ^t^I?!, 4-^2 - s| ^^2 > ^Sfl-D2, for i? large enough, 
and so 



,p ^2— p ,p p 2— p 



-Fi?, — ^ p Di > — -p-Di, Fr — —p — ■ — D2 > 777p-D2, 



which implies that 

< ^DiD2 < 0, as i? ^ +00, 

a contradiction. Therefore, tji and sr are uniformly bounded. Then by (j2.2ip 

and Fji ^ as i? — > 00, we get that 

lim {\tR-l\ + \sR-l\)^Q. 
Note that {tRUR, SrVr) G Af, we see from (|1.16p that 

A < I{tRUR, srVr) i- ( 4 / iVwflp dx + sl / \\/vr\^ dx 

V JrN JrJV 
1 / Af-2 N-2 \ 

= ^(4Mr^ + 4M2"^)^~/^- 

iV-2 iV-2 

Letting i? ^ +00, we get that A < ^(^1 ^ + /I2 ' 

On the other hand, for any {u, v) £ Af, we see from /? < and (|1.14p that 

/ \Vu\'^ dx < f \u\^P dx < fiiS-P ( f iVupdx) , 
and so iVupdx > fi'^S'^^^. Similarly, iVwpdx > fi^^S'^^^. 

N-2 N-2 

Combining these with ( [TTg)) . we get that A> jrip^ " +^2 ^ jjence, 

1 / N-2 N-2 \ 
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Now, assume that A is attained by some G A/", then \v\) G and 

J(|u|, \v\) = A. By Lemma [231 we get that \v\) is a nontrivial solution of 
(jl.l3p . By the maximum principle, we may assume that u > 0, u > and so 
uPyP dx > 0. That is, 

/ \Vu\'^dx<fiif \ufPdx<fiiS-p(f \Vufdx 
Therefore, it is easy to see that 

A = I{u,v) = - y^^dVup + \^vf)dx > - [^H^+^^2^) s""/', 
which is a contradiction. This completes the proof. □ 

Proof of (2-1) in Theorem 11.61 . This proof is similar to the proof of 
Theorem ll.il Since /3 > 0, by Lemma [2T] equation (jl.9p has a solution {ko,lo). 
Then {^/koUe^y, \/ToUe,y) is a nontrivial solution of (|1.13p and 

A<I (^Vk'oU.^y, VfoU.^y) = ^(fco + lo)S''/^. (2.23) 

Assume that P > {p — 1) max{/xi, /i2}. Let {(it„,w,i)} C A/" be a minimizing 
sequence for A, that is, /(m„,w„) — ^ A. Define c„ = (/rn |w„P'' dx) (i„ = 
(/rn |wnP^c?x)^^^, we have 

Scn< I \Vu^fdx=[ ml"n|'^ + /3knrk„rrfa;<^ic^ + /3</2dP/2^ 

Sdn< f \VVr,\^dx= f fi2\Vn\^P +P\u^nv^\Pdx<fi2dl+l3cPj^dP/\ 
JR" JRW 

This means 

S{Cn + dn) < NI{Un, «„) < {ko + /q)^"^/' + o(l), 

/zicP-i + f3cPj^-'dP/^ > S, PcPj^dPj^-' + ii^dP-^ > S. 

Similarly as in the proof of Theorem 11.11 we see that c„ — ^ koS^^^"^ and 
dn — IqS'^^^^^ as n +oo, and 

NA = lim iV/(u„, w„) > lim S'(c„ + d„) = (fco + lo)S^^'^. 
This implies that 

^ - ^(fco + ?o)5^/' = I{Vh>Ue.y, VkUe^y), (2.24) 
and so {VkoUe,y,\/ToUe,y) is a positive least energy solution of (jl.lSp . □ 
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To finish the proof of Theorem ll.61 we need to show that (I1.13P has a positive 
least energy solution for any < (3 < {p — 1) max{^i, /i2}- The following proof 
works for all /3 > 0. Therefore, we assume that (3 > 0, and define 

A' -.^ inf I{u,v), (2.25) 

where 

Af' ■.= {{u,v)eD\{{0,Q)}, I (IViip + |Vz;n 

I- .ON 



(/iilupP + 2p\u\P\v\P + M2bP^) = O}. (2.26) 

r J 



Note that jV C A/"', one has that A' < A. By Sobolev inequality, we have 
A' > 0. Define B{0,R) := {a; € : |a;| < R} and i7(0,i?) := H^{B{0,R)) x 
H^{B{0,R)). Consider 

-Au = ^i|u|2P-2u + /3|u|P-2u|z;|P, x £ 5(0, i?), 

-Av ^ H2\v\^P^^v + l3\v\P-^v\u\P, xeB{0,R), (2.27) 
e H^{B{0,R)), 



and define 



where 



A'(i?) inf I(u,v), (2.28) 

(u,i))S7V'(fl) 



AA'(i?) {u, v) e i/(0, i?) \ {(0, 0)}, / (|Vu|2 + \Vv\^) 
[ Jb{o,r) 

i^i^\ufP + 2p\unvf + ^i2\vfn = ol. (2.29) 

B{O.R.) J 

Lemma 2.6. A'{R) = A' for all R>0. 

Proof. Take any Ri > R2. By 7V'(i?2) C Af'{Ri), we have A'{Ri) < A'(i?2). 
On the other hand, for any {u, v) G Af'{Ri), we define 

/ N-2 JV-2 \ 

Ri\ ' f Ri \ f Ri\ ' f Ri 



(.,(.),.,(.)) :=(^(^-j -1^]^- 

then it is standard to see that (ui, vi) G M' {R2), and so 

A'{R2) < I{ui,vi) ^ I{u, v), V(u, v) e N'{Ri). 

That is, A'(i?2) < A'(i?i) and so A'{Ri) = A'(i?2). 

Obviously, A' < A'{R). Let (ti„,u„) S A/"' be a minimizing sequence of A'. 
Moreover, we may assume that m„,w„ G Hq{B{0, Rn)) for some i?„ > 0. Then 
{un,Vn) e M'{Rn) and 

A' = lim /(-itn,!;,!) > lim A'(i?„) = A'(i?). 
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Therefore, A'{R) = A' for all R>0. □ 
Let < £ < p — 1. Consider 

'-Au = ^i|u|2p-2-2e^t _^ (3\u\P-^-^u\v\P-^, X e 5(0, 1), 
-Av = H2\v\^^^^-'^^v + l3\vY'-'^-^v\uY'-'', xe -8(0,1), (2.30) 



and define 



A, := inf /,(u,z;), (2.31) 



where 



h{u,v)=\ [ i\Vuf + \Vv\') 

^ •■'5(0,1) 
1 



2p- 2e 

and 



B(0,1) 



:= \{u,v) e H {0,1) \ {{0,0)}, H,iu,v) := / {\Vu\^ + \Vv\') 

[ "'-8(0,1) 

iliilufP-'' + 2l3\u\P-'\vr' + ^i2\vfP-'n = I. 

S(0,1) J 

Lemma 2.7. -For any < e < p — 1, there holds 

A^<mml inf /^(m, 0), inf 4(0, u) 

[(«,0)e7V,' (0,i;)e7V^ 

Proof. Fix any < e < p — 1. Recall that 2 < 2p — 2£ < 2* , we may let Ui be 
a least energy solution of 

-Au = u e H^{B{0, 1)), 

Then 

/e(ui, 0) = ci := inf /e(u,0), /^(O, M2) = C2 := inf 4(0, w). 

(«,0)GA^^ (0,t))eA/',' 

The following proof is inspired by 4 . For any s e K, there exists a unique 
t{s) > such that {t{s)ui,t{s)su2) G A/"^. In fact, 

. ^ /b(o,i)(|V-iI^+^^|V"2P) 

_ p'ci + S^p'c^ 

" p'ci + VC2 + \S\P-' 2/3|ui|P-=|M2|f-" ' 
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where -p' = Note that t{0) = 1. Recah that 1 < p - e < 2, by direct 

computations we have 

t'{s) (p-£)/5(oi)2/3Kr-Kr^ 

lim — 



s^o |s|P-2-es {2p - 2e - 2)p'ci 



that is, 



(p-E)/mni|2^l«ir'l«2|'"' 



and so 

This imphes that 

(2p-2£) f„,„ 2^|wi|P-^|w2|P-^ 

.i--'«"'^^':-''"'-''-|.r-(i.oa)),...^o^ 

2ci 

Therefore, we see from 1/2 - 1// = l/(2p - 2e) > that 

< h {t{s)ui,t{s)sU2) 

= L'ci + |s|2p-Vc2 + \sr' [ 2(3\u,\P-'\u2rA 

V2 P'J Jb(os) 

< ci = inf /e(u,0) as \s\ > small enough, 

(■u,o)e7V^ 

By a similar argument, we have < inf h(0,v). □ 

(o,i>)e7V^ 

Recall uj^. in the proof of Theorem ll.6l -fl). by the same argument as Lemma 
12.71 we have 

A'<mini inf /(u,0), inf l(0,v)\ = mm h(uj^„0), l(0,uja.)\ 

l^^'^S^/'Y (2.32) 

Theorem 2.1. for any < e < p — 1, \2.30\) has a classical least energy 
solution {ue,Ve), andue,Ve are both positive radially symmetric decreasing. 
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Proof. Fix any 0<£<p— 1, itis easy to see that > 0. For {u, v) S Af^ 
with u > 0,v > 0, we denote by {u*,v*) as its Schwartz symmetrization. Then 
by the properties of Schwartz symmetrization and /3 > 0, we have 

{\\/u*f + \\/v*f)< [ {fii\u*fP~^' + 2(3\u*\P-'\v*\P~' + fi2\v*fP-^'). 

5(0,1) JB{0,1) 

Therefore, there exists < t* < 1 such that {t*u*,t*v*) £ Af^, and then 
I,{t*u*,t*v*) ^(l - ^ \ ) {t*f f (|Vii*|2 + |Vt;*n 



2 2p - 2e 



S(0,1) 



Therefore, we may take a minimizing sequence (u„,u„) G of such that 
(u„,u„) = and /e(M„,w„) — )■ A^. We see from (|2.33l) that u„,w„ are 

uniformly bounded in Hq{B{0, 1)). Passing to a subsequence, we may assume 
that Un Us,Vn Ve wcakly in Hq{B{0,1)). By the compactness of the 
embedding H^{B{0, 1)) ^ L^p-^^B{0, 1)), we have 



Js(o,i) 

2p-2e , , , 2p-2e ^ 

lim Ie[Un,Vn) = Ae > 0, 



— e n->oo p — 1 — £ 

which imphes that ^ (0,0). Moreover, > 0,W£ > are radially 

symmetric. Meanwhile, /g^g -^^(|VMep + |Vwep) < Jiirn^ /b(o.i)(|Vw„P + |Vi;„P), 
so 

B{OA) JB(as) 
Therefore, there exists < < 1 such that {t^Ue,t^v^) € J\f^, and then 

< hm f (|Vw„p + |V«„n 

5(0,1) 



^2 2p-2e 

lim Ie{Un, Vn) = A. 



Therefore, ~ 1 and {ue,Ve) E Af^ with I{ue,Ve) = A^. Moreover, 

/ (|Vu,|2 + |Vt;,n= lim / (|Vw„|2 + |Vw„n, 
Jb(o.i) "^°°Jb(o.i) 
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that is, Un — > and i;„ — > strongly in Hq{B{0, 1)). There exists a Lagrange 
multipUer 7 e M such that 

I'^iu„v,)--fH'^iu„v,) = 0. 

Since I'^{Ue,Ve){Ue,Ve) = He{Ue,Ve) = and 

H'^{ue,v,){ue,v,) = {2 + 2e-2p) f {fnulP-^' + 2(3uP-'vP-' +fi2V^/-^') < 0, 

Js(04) 

we get that 7 = and so I'^{ue,Ve) = 0. By Lemma [2771 we see that ^ and 
Ve ^ 0. This means that (m^, Wg) is a least energy solution of (|2.28p . Recall that 
WejWg > are radially symmetric non- increasing. By regularity theory and the 
maximum principle, we see that u^, > in i?(0, 1), u^, S C^(-B(0, 1)) and 
are radially symmetric decreasing. □ 

Completion of the proof of (2) in Theorem 11.61 . Recall (I2.29p . for any 

e A/''(l), there exists > such that {t^u,t^v) e Af^ with — >• 1 as 
e ^- 0. Then 

limsupAe < \imswp I^{t^u,ti;v) — I{u,v), y{u,v) G A/''(l). 

e-i-O £-i-0 



By Lemma |2. 61 we have 

limsupAe < A'(l) = A'. (2.34) 

e-s-O 

By Theorem l2.1[ let {u^, w^) be a positive least energy solution of (|2.30p . which is 
radially symmetric decreasing. By I'^{ue,Ve){ue,Ve) — and Sobolev inequality, 
it is easily seen that 



p — e — 1 



Ae= f (|V7.eP + |Vi;,n>Co, V0<£<^, (2.35) 

JB(0,1) ^ 



where Cq is a positive constant independent of e. Then are uniformly 

bounded in Hq (-6(0, 1)). Passing to a subsequence, we may assume that uq 
and Ve vq weakly in Hq{B{0, 1)). Then (uo,wo) is a solution of 

-Au = /ii|u|2p-2y ^_ p\u\P-^u\v\P, X G B{0, 1), 
~Av = fi2\v\^P~^v + (3\v\P-^v\u\P, xeB{0,l), (2.36) 
u,veH^{B{0,l)). 

Assume by contradiction that ||ue||oo + 1 1 1 1 00 is uniformly bounded, then 
by the Dominated Convergent Theorem, we get that 

lim / u^P"^' = [ u^P, hm / ^ f ^2p ^ 

'^^"Jb(0,1) J B(a,l) ^^'^Jb(QA) "'-8(0,1) 



lim 

e->0 



uP^^yP-' = 
-8(0,1) -'5(0,1) 



X,Q UQ. 
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Combining these with I'^{u^,v^) = I'{uo,vo) = 0, it is standard to show that 
Ue — >■ uo and — >■ vq strongly in Hq{B{0,1)). Then by (|2.35p . we see that 
(ito,uo) 7^ (0,0). Moreover, uq > 0, uq > 0. We may assume that uq ^ 0. By 
the strong maximum principle, mq > in -6(0, 1). Note that 2p = 2*. Combining 
these with Pohozaev identity, we have 

0< [ i\Vuo\^ + \Vvo\^){x-iy)d<7 = 0, 

JdB(0,l) 

a contradiction. Here, v denotes the outward unit normal vector on dB{0, 1). 
Therefore, HueHoo + ||we||oo -> oo as e 0. We will use a blowup analy- 
sis. Note that u^iO) = max Up(x) and v^iO) = max v^ix), we define := 

' 5(0,1) S(0,1) ^ ^ 

TOax{ue(0), ^^(O)}, then — )■ +oo. Define 

C/,(x) = i^7V(i^7"^a;), V,{x)^K-\,{K-''^x), a,=p-l-e. 
Then 

1 = max{C/e(0),K(0)} = maxi max Ue(x), max K(a;) I (2.37) 
and [/g , Vs satisfy 

= /ii[/2p-2^-i + fiUP-'^-^VP-" , X e 5(0, iC^"-), 

= /izK^P-^^-i + /3y/-i-^J7P-% a; G B(0, /sT^"-)- 

Since 




we see that {(C/^, K)}„>i is bounded in D^-^{M.^) x D^'^^igW) ^ gy elliptic 
estimates, for a subsequence we have (C/g, V^) — > (?7, 1^) G L* uniformly in every 
compact subset of as e ^ 0, and (C/, V) satisfies PTT^ . that is /'(J7, y) = 0. 
Moreover, t/ > 0, > are radially symmetric non-increasing. By (|2.37p we 
have ([/, V) ^ (0, 0), and so (t/, V) € A/"'. Then we deduce from ([OI)) that 

A' < /([/, v^) = - ^) (I vt/p + ivyp) 

<liminf fi--^) / {\VU,\^ + \VVe\^)dx 



0^2 2p - 2e 
< liminf []-- ^ \ ) / (|Vu,|2 + |Vi;endx 



0^2 2p - 2e 
= liminf < A'. 



B(0,Kfn 

r 

B{0,1) 



This implies that I{U,V) = A' . By ((O^ we have that ^ and ^ 0. 
By the strong maximum principle, U > and > are radially symmetric 
decreasing. We also have {U, V) € N, and so I{U, V) > A> A' , that is, 

I{U,V)= A = A', (2.38) 
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and {U,V) is a positive least energy solution of (|1.13p . which is radially sym- 
metric decreasing. 

Finally, we show the existence of (fc(/3), /(/?)) for /3 > small. Recall (|2.ip - 
(E^, We denote ai{k,l) by ai{k,l,P) here. Define k{0) = fi^'^ and Z(0) = 
, then a,(fc(0),?(0),0) = 0,i = 1,2. Note that 

9feai(fc(0),/(0),0) = (p- lVifc(0)P-2 > 0, 
dia2{k{0),l{0),0) = {p-l)fi2l{0r-^ > 0, 
diai{k{0),l{0),0) = dka2{k{0),l{0),0) = 0, 

which implies that 

/ afcai(fc(0),/(0),0) 9,ai(fc(0),Z(0),0) \ 
^ afca2(fc(0),/(0),0) dia2{k{0),l{0),0) J > 

Therefore, by the implicit function theorem, k{(3), 1{I3) are well defined and class 
on i ~ 132, 1 32) for some /Ja > 0, and ai{k{l3),l{l3), /3) = 0, i = 1, 2. This implies 
that {y/k(J3)Ue,y, y/l{(3)Ue,y) is a positive solution of pTHl) . Note that 2p = 2*, 
This implies that 

/ \ N-2 N-2 

hm (fc(/3) + ?(/3)j = fc(0) + ?(0) = /ii ^ +M2 ^ , 
that is, there exists < /3i < ^2, such that 

>min|/i, ^ , ^ | , V/3e(0,/3i). 

Combining this with ^ft^^ and (|2?32l) . we have 



/({/, = A' = ^ < I{^k{p)U,.y, VmUe.y), V /3 e (0, 



that is, {\J k{f3)Ue,y, \Jl{(3)Ue,y) is different positive solution of (|1.13p with re- 
spect to (J7, V). This completes the proof. □ 

Before ending this section, we need to study the following properties of (C/, V) 
obtained in Theorem 11.61 



Proposition 2.1. Assume that (3 > 0. Let (U, V) be a positive radially symmet- 
ric least energy solution of il.l3\) obtained in Thoerem \l.b\ Then there exists 
C > such that 

U{x) + V{x) < C{1 + \S/U{x)\ + \\/V{x)\ < C{1 + 

Proof. Define the Kelvin transformation: 



22 



Then U*,V* G D^^^{R^) and {U*,V*) satisfies the same system (fTT^ . Then 
by a standard Brezis-Kato type argument ([8]), we see that U* ,V* G i°°(R^). 
Therefore, there exists C > such that 

C/(x) + < C|xp"^. (2.39) 

On the other hand, note that U, V are radiahy symmetric decreasing. We also 
have U,V e L°°(M^), and so 

U{x) + V{x)<C{l + \x\)^-^. 

Moreover, standard ehiptic regularity theory implies that U,V ^ C^(R^). We 
write t/(|a;|) = U{x) for convenience. Then 

and so for any i? > 1, we see from (|2.39p that 

R^-^\Ur{R)\ < \Ur{l)\+ j\''-\tJiiU^'-^+fiU^''^-^V^''^)dr 

/ + 00 

Therefore, it is easy to see that \VU{x)\ < C(l + \x\)^-^ for some C > 0. 
Similarly, \VV{x)\ < C(l + \x\)^-^ . □ 

3 Proof of Theorem [TS] 

In this section, we assume that — Ai(f2) < Ai < A2 < 0. Recall the definition of 
B in (11.51) . since 




1 = 1,2, 



it is standard to see that B > 0. As has been pointed out in Section 1, by [9] 
the Brezis-Nirenberg problem (|1.3I) 

-Au + AjU = ueH^i^l) 

has a positive least energy solution u^^ e C^(ri) n C(il) with energy 

h i^^y^'-^'"" ^ - \ /„<iv..j'+A.<)- 1 /^,..< 

<-if.-'^S»'^ . = 1,2. (3.1) 

The following lemma is very important. In the proof we need the assumption 
Ai, A2 < 0. 
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Lemma 3.1. Let /3 < 0, then 

( 1 N — 2 1 N — 2 ^ 

B < min ji?^, + -^-^5^/2, B^, + A j . 

Proof. The idea of this proof comes from [T7], but some details are more 
dehcate. Let /? < 0. Let to > such that 

^B,,t2-^B,,i2^ + lW2r^5^/2<0, Vt>io. (3.2) 

Since € C(ri) and = on 9f2, there exists 

B{yo, 2R) := {x : |x - yo| < 2i?} C f^, 

such that 



JJ-2 V"' /^Ai + Ai(f7)V-i 



5:=„max^,.,,<nnn<|(^) , ( , M ^ (3-3) 



Let -0 e Cl{B{yo,2R)) be a function with < V < 1 and i/i = 1 for 
k ~ yo| £ R- Define = tl>Ue,yoi where Ue,yQ is defined in and (|1.16p . 

Then by |9] or [38l Lemma 1.46], we have the fohowing inequahties 



|Vz;,|2 = 5^/2 + 0(e^-'), / \vef = 5^/' + 0(£^), (3.4) 

JO 

/ |«.|2>C£2 + 0(e^-'). (3.5) 
Jn 

Moreover, since > 5, we have 



N/2 



B(yo,2R) JB{a,2R) 



+ a; 



<Ce'"^[\n— + l\=o{e^). (3.6) 



Since supp{v^) C i?(2/o, 2i?), by p.3p we have for t,s > that 



n 



< I i,,u^l + ^s'^ I ^,v'/, (3.7) 
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and so 

= /(t)+.9(s). (3.8) 
By (133)-(in3D, it is standard to check that (cf. [51155]) 

max5(s) < -^(^2/2)^^6'^/^ for e small enough. (3.9) 

By (jS.ip we see that 

Combining these with p. 21) . we get that 

/(i)+3(s) <0, Vt>to, s>0, 
and so it follows from (I3.8P that 

max _E(iMM, , sUe) = max Eituu,, , sv^). 

t,s>0 ^ 0<t<to,s>0 

Define 

Then there exists a unique s(e) > 0, such that g'^{s{e)) — with 



= 1 



/nM2W?^da; V Ai(17)/ C^l.ao da; 

A2 \ 5^/2 + 0(e^-2) 



Ai(l]); M2^^/2 

> ( 1 + ,'1^, ) =: Sq^~^, for e small enough. 
2/^2 V Ai(i2)/ 

Therefore, since is increasing for < s < s{e), for any < s < sqj we have 
ge{s) < 5e(so) and so E(tu^^,sv^) < E{tUf^^, sqv^). That is, 

max E(tUu, , sve) = max E(tUn,,sVp). (3.10) 

t,s>0 ^ 0<t<to,s>so ^ ' 
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For < t < to, s > sq, we see from p.6p that 



Ml 

n 



and so 



Jn 



=:fi{t)+9i{s). (3.11) 

Note that maxt>o fi{t) = /i(l) = B^^. By ((3^ - ([33)) and A2 < 0, it is easy to 
show that 

maxgi(s) < — u, ^ " for e small enough. 
Combining these with p.lOp and p.lip . we obtain that 

max E{tUaj^, sVg) ~ max E(tUn,-^, svi^) 

t,s>Q Q<t<to,s>so 

< max/i(t) + max 51(5) 

t>0 s>0 

1 Af-2 

< B^,, + —^2 ' foj. g gj^all enough. (3.12) 

Now, we claim that there exists t^Se > such that (tjU^^jSeUe) G 7W. 
Similarly as p.7p . we have 

O JO 



(Ai(17) + Ai)m2 Jn 

< / fiiu'^^dx / ^2v1^ dx. 
Jn Jn 



For convenience we denote 



^1 = / A^i"'^ dx, D2= f 
Jn Jn 

Ds^ [ fi2V^/dx, ^4 = / (iVwep + A2V,') dx. 
Jn Jn 
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Then D2 < and D1D3 — > 0. Furthermore, {tUf_,^,sv^) G M for some 
s > is equivalent to 

t^-PDi ^tPDi + SPD2, s^~PD4,^ sPD3 + tPD2, s,t>0. (3.13) 

Note that I < p = jf^ < 2, by sP = (t^-p _ tP)Di/D2 > we have i > 1. 
Therefore, p.l3p is equivalent to 

Since /3(1) > and hm /3(i) < 0, (I3.14p has a solution t > 1. Therefore, 

p.l3p has a solution > 0, > 0. That is, {t^u^^ , s^Vg) G M and from (|3.12p 
we get 

1 — 

t,s>0 iV 

_ iV-2 

By a similar argument, we can also prove that B < B^.^ + j^p^i ^ S^^'^. By 
and dsn]), we have 

f 1 iV-2 1 N-2 ^ 

A>max|i?^, ^ i?^, + -//i ^ 

This completes the proof. □ 

Lemma 3.2. Assume that /? < 0, then there exists C2 > Ci > 0, such that for 
any (w, v) G Ai with E{u, v) < A, there holds 



Ci < / \u\'^Pdx, / \v\^Pdx < C2. 
Jn Jn 

Proof. This follows directly from 

E{u,v) < A a.nd □ 

Lemma 3.3. Let Un u,Vn ^ v in i?o(i^) as n ^ 00, then passing to a 
subsequence, there holds 



lim / i\Un\P\Vn\P-\u„~u\P\Vn~v\P~\u\P\v\P) dx^O. 
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Proof. Note that 2p = 2* , we have 

Un u, w„ — > w strongly in L'(r2), \f < q < 2p, 
Un u, Vn V weakly in L'^P{n). 

Fix any t E [0, 1]. First, we claim that 
\u„-tu\P-^{un-tu)\vn\P ^ {1 - t)P~^ \u\P-^u\v\P weakly in L^{n). (3.15) 

Since the map h : L'^^ifl) -> Li^/'J^{n) with h{s) = \s\'^^~^s is continuous, we 
have 

|u„ - tu\P~^{un - tu) (1 - tY-^\uY'-^u strongly in L9(17), V < g < 
\vn\P \v\^ strongly in Li{n), V < g < 2. 

Then for any 1 < g < 2p~i ' Holder inequality, we have 

\un - tu\P-'^{un - tu)\vnY' " Wn - - tu)\v\P strougly in i«(rJ), 

|u„ - tu\P-^{un - tu)|wr - (1 - t)P-^\u\P-^u\v\P ^ strongly in L'^{n), 

and so for any 1 < q < 2p-i ; 

|m„ - tul^^^K - iu)|i;„r ^ (1 - strongly in Li{n). 

2p 

Since |u„ — iu|^'^^(u„ — iit)|w„|^ is uniformly bounded in L^p-^ (f2), passing to 
a subsequence, we may assume that |u„ — — <u)|w„|p w weakly in 

L^{n). Then for any (p e C^(R^), we have 

wcp^ lim / |M„-twr2(M„-iu)|t;„|f(^= / {l-t)P-^\u\P-^u\v\Pip, 



which implies that w = (1 — t)P ^\u\p '^u\v\p, that is, p.lSp holds. Similarly, 
we can show that 

|u„ - u\P\vn - tv\P^'^{vn - to) ^ wcakly in L^(fi). (3.16) 

Therefore, by p.l5p - p.l6p . Fubini Theorem and the Dominated Convergent 
Theorem, 



{\Un\^\Vn\^ - \Un - u\P\Vn - v\P) dx 
d 

In Jo 

1 /• 1 



i\Un - tu\P\Vn\P + \Un " u\P\v„ - tv\P) dt dx 

1 

2/ 



n Jo 



nl 

+ P / \Un — u\P\Vn — tv\P~'^{Vn — tv)v dtdx 



n Jo 
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\Un ~ tu\P^ {Un — tu)\Vn\^udx dt 

/o Jn 

r-l 

,|P-2 



+ P / \Un — u\^\Vn — tv\P (Vn — tv)v dx dt 

Jo Jn 

[ [ {l-t)P-^\u\P\v\Pdxdt= [ \u\P\v\Pdx, 
Jo Jn Jn 



as n — )■ oo. 

This completes the proof. □ 

Proof of Theorem 11.31 for the case /3 < 0. The main idea of the proof is 
similar to the proof of [ITl Theorem 1.3-(1)] in case = 4, but as we will see, 
some new ideas are needed. Assume that /3 < 0. Note that E is coercive and 
bounded from below on M.. Then by the Ekeland variational priciple (cf. [31]), 
there exists a minimizing sequence {(un,v„)} C M satisfying 

E{un, w„) < min I B + i, A I , (3.17) 



-B(w,w) > £;(u„,u„) - -||(m„,w„) - (u,w)|j, V(u,w)eX. (3.18) 
n 

Here, ||(m,w)|| := (^(|Vu|2 + \Vv\'^) dxf/'^ is the norm oiH. Then {(w„,?^n)} 
is bounded in H. For any (<p, (/>) G H with \\ip\\, ||(/)|| < 1 and each n £ N, we 
define the functions ft,„ and g„ : R'^ — > R by 



h„(t,s,l)^ / |V(ii„ + + sii„)| + Ai / |w„ + t(^ + su„| 
Jsi Jo 

-/^l / \Un + tip + SUnl^P ~ P \Un+tip + SUn\''\Vn+t(t) + lVn\P, (3.19) 

Jn 

and 

gn{t,s,l)^ / |V(?;„ +t<?!) + ;v„)p + A2 / |?;„ + t<?!) + ;w„p 
Jn Jn 

-/^2 / \Vn + t(t) + IVnl^P - 13 \Un + tip + SUn\P\Vr, + t(t) + lVn\P . (3.20) 

Jn Jn 
Let = (0,0,0). Then e Ci(R3,M) and /i„(0) = g„(0) = 0. Further- 



more, 



^(0) = -(2p - 2)/ii ^ |,.„|2P - (p - 2)/3 1^ \u^\PK\P, 

^(0) = ^(0) = -p/?^^ |^.„nz;„r dx, 

(0) = -(2p-2)/i2 / |i;„|2P-(p-2)/3 / 
Jn Jn 



dl 



Define the matrix 



^(0) ^(0) 
^(0) %^(0) 
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Since /? < and (it„, v„) € A^, we have 



M2 / |w. 
In 



|2p 



and it follows from Lemma 13.21 that 



det(F„) > (2p-2)^ / i\Vun\' + Xi\unn / i\VVn\' + X^') 



> cs^ 



|2p 



2p I > C > 0, 



(3.21) 



where C is independent of n. By the implicit function theorem, functions s„(i) 
and are well defined and class on some interval (— ^n, +^n) for J„ > 0. 
Moreover, s„(0) = /„(0) = and 

hn{t, Sn{t), ln{t)) = 0, Qnit, Sn{t),ln{t)) = 0, {-6, +6). 

This implies that 

4(0) = (^(0)^(0) - %(0)^(0)) , 

^C(O) = ^ (^(0)^(0) - ^(0)^(0)) . 

While, since «„)} is bounded in H, it is easy to see that |^^(0)|, |^^(0)| < 
C, where C is independent of n. From Lemma 13.21 we also have 

< C. 



di ^ ' 



Hence, combining these we have 

14(0)1, lC(o)l<c, 



(3.22) 



where C is independent of n. 

Define (/3„,t := w„ + + Sn{t)un and 0„,t := Vn + t<j) + l,i{t)vn, then 
{(pn,t,(f>n,t) G for < e (— (Jriif^ji). It foUows from p.lSp that 

E{(Pn,t, (t)n.t) - E{Un,Vn) > -- \\{t(p + S„ (t)M„ , t0 + /„ (t) W„ ) 1 1 . (3.23) 

n 

Note that £"(«„, '(;„)(u„, 0) = £^'(u„, u„)(0, u„) = 0. By Taylor Expansion we 
have 

E{'Pn,t,4>n,t) - E{Un,Vn) = S'(u„,U„) {t(p + S„(t)u„,t0 + /„(i)v„) + r(?l, i) 

= tS'(u„,v„)(^,(^) +r(n,i), (3.24) 
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where r{n,t) = o{\\{tip + Sn{t)un,t<f) + ln{t)vn)\\) as t — > 0. By p.22p we see that 



hm sup 



ip H —Un,(p-\ —Vj 



< C, (3.25) 



where C is independent of n. Hence, r{n,t) = o(t). By ((3?23l) . ((3?24| . ((3?25l) 
and letting i — > 0, we get that 

C 

|£;'(u„,i;„)((^,(/))| < — , 
n 

where C is independent of n. Thus, 

hm E'{un,vn)=0. (3.26) 



Since {(u„,ti„)} is bounded in i7, we may assume that (u„,u„) {u,v) 
weakly in H . Passing to a subsequence, we may assume that 

Un u, Vn V, wcakly in L^^(ri), 
Wnl^-^u,, ^ \u\'^-'u, - \v\'^~'v, weakly in L^P/i{n), 1< q < 2p, 

Un — ?> u, Vn ^ V, strongly in L^(0). 

Thus, by (13.26^ we have E'(u, v) = 0. Set a;„ = Un — u and cr„ = Vn — v. Then 
by Brezis-Lieb Lemma (cf. |38]), there holds 

Wn\Z = HZ + + "(1)' = K + \^^t + (3-27) 

Note that {un,Vn) £ M and E'{u,v) = 0. Combining these with p.27p . 
Lemma [331 we get that 



|Vc^„|2„ / (^^|^„|2p + ^|^^|P|^^|P) = o(i)^ (3.28) 
Jn 

|Va„|2 _ f {f,^\an\^P + /3\ujnn<Jnn = o{l) , (3.29) 

E{Un, Vn) = i?(u, w) + /(Wn, (T„) + o(l). (3.30) 

Passing to a subsequence, we may assume that 

lim / |Vw„p^6i, lim / |Vct„P = 62. 

Then by ([^^ and ((X^ we have /(w„,cr„) = ;^(6i + 62) + o(l). Letting 
n — >■ +00 in (I3.30p . we get that 

0<E{u,v)<E{u,v) + ^{bi+b2)^ lim E{un,Vn) = B. (3.31) 

iV n— >-+oo 

Case 1. u = 0,v = 0. 
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By Lemma [5?^ (P??7|) and ([53T|) . wc have < &i < +cxi and < 62 < +00, 
and we may assume that both a;„ ^ and an ^ for ri large. Then by p.28p 
and p.29p we have 

/ Mil^nP^ / A^2|(T„|2P f i^^Ji^^jP^j > 0, for n large. 

Then by a similar argument as that in the proof of Lemma l3.11 for n large, there 
exists tn,s„ > such that (<na;„,s„cr„) G M. Up to a subsequence, we claim 
that 

lim (|t„-l| + |.s„-l|) = 0. (3.32) 



n— f +00 



This conclusion is obvious in case = 4 and p — 2 (see [U]), but it is not 
trivial in our general case N > 5 here. Denote 

Bn,l= [ IVuJrf^h, B„,2= / |Vcr„|2^62, 

2p 
n I 5 

n 



JSl Ji 
= |/?| / \uJnf\cTn\ 

Jn 



Passing to a subsequence, we may assume that Cn,i -> ci < +cxd, C„,2 — >■ C2 < 
+00 and Dn-^ d< +00. By ((3?M)) - ((3?29)) we have 

ci = 61 + > 61 > 0, C2 = 62 + > 62 > 0, (3.33) 

i2B„,i = _ tP^slDn, slBn,2 = slPCn,2 - tlslD^. (3.34) 
This implies that 

V > 7; ^ — > 0, s/ > — ^ — > 0. (3.35) 

'^nS Ci 0„,2 C2 

Assume that, up to a subsequence, tn +00 as n — > 00, then by 

tf^Cn,! — t^Bn^l = s'^Cn,2 — S^^n,2, 

we also have s„ — >■ +00. Then 



= lim Di = lim 



n—>oo 



= lim (C„,i - t^2^'B„,i)(C„.2 ~ sl-^PB.n,2) 

n— >-oo 

= C1C2 = (61 + d){b2 +d)> d^, 

a contradiction. Therefore, i„,s„ are uniformly bounded. Passing to a sub- 
sequence, by p.35p we may assume that i„ — > too > (61/ci) ^p-s > and 
Soo > (62/132)^^^ > 0. Then we see from (I3.34p that 



'2- 
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If d = 0, then c,; — bi, and so too = Soo = 1- That is, p.32p holds. Now we 
consider the case d > 0. Define f{t) = t^ci — t^^Pbi, then for t > {bi/ci)^p^ , 
we have 

fit) = pcitP-^ - (2 - > (2 - p)t^~P{cit^P-^ - bi) > 0, 

that is, / is increasing with respect to t > (bi/ci) ^p-^ . If ico < 1, then 

S^d = /(ioo) < /(I) = Cl - 6l = rf, 

that is, Soo < 1, and we see from (|3.33p that 

fP r-i — t'^^Phi sP ro — t'^^Ph'^ 
^2 ^ ^ooC^^o^ So^C^So^ ^ _ ^2-2p^^)(^^ _ ^2-2p^^) 

5oo t-oo 

= (d + 6i - tl^^Pbi){d + 62 - s^'''fo2) < d', 
a contradiction. If > 1, since 1 > {bi/ci)^p^ , we have 

S^d = /(ioo)>./(l) = Ci-6i=d, 

that is, Soo > 1, and so 

d^^{d + bi- t^o^^Pbi){d + 62 - > d', 

a contradiction. Therefore, too = Soo — 1 and p.32p holds. This implies that 

-^{bi+b2) = lim I{uj,i,an)= lim /(t„a;„, s„cr„) > A. 

iV n— >-+oo n— f+CJO 

Combining this with p.3ip we get that B > A, a contradiction with Lemma 
13.11 Therefore, Case 1 is impossible. 

Case 2. u ^ 0, w = or w = 0, w ^ 0. 

Without loss of generality, we may assume that u ^ Q,v = 0. Then 62 > 0. 
By Case 1 we may assume that 61 = 0. Then lim„_j.+oo |w„|p|(7„|^ = 0, and 

/ |Va„P= / mWnl^" + 0{l) < mS-P ( [ \Van\A +0(1). 

Jn Jn \Jn / 

iV-2 

This implies that 62 > M2 -S"^'^- Note that u is a nontrivial solution of 
-Au + Aiu = we have from (|3T|) that £;(u,0) > S^,. By (|33T|) we 

get that 

1 1 _ ^^-2 

B>B^,+-b.2>B,, + -f,, ^ 
a contradiction with Lemma [3. II Therefore, Case 2 is impossible. 
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Since Cases 1 and 2 are both impossible, we have that u ^ 0, w ^ 0, that 
is, {u,v) £ M. By ((33T1) we have E{u,v) = B. Then e M and 

£'(|w|,|w|) = B. By Lemma [^31 is a solution of (jl.ll) . Then, using 

the maximum principle, we see that > in 17. Therefore, \v\) is a 

positive least energy solution of p.ip . This completes the proof. □ 



It remains to prove Theorem 11.31 for the case /3 > 0. Assume that (3 > 0. 
Define 

B := inf max E(h(t)), (3.36) 
herte[os] 

where T ^ {h e C([0, : /i(0) = (0,0), E{h{l)) < 0}. By dH]), we see that 

for any e i/, ^ (0,0), 

maxEitu, tv) = E(tu vU, vv) 
t>o ' 



N 



= / (MikP^ + 2/3knz;r+M2|t'|'^), (3.37) 
where <„^u > satisfies 

/o(Mi|"pP + 2/3|u|P|i;|?' + M2kP^)' ■ 
Note that {tu.vU,tu.vv) € TM', where 

M' := {(u,v) e i/\{(0,0)}, G(u,w) ^ (jVup + Aiu^ + |V?;p + Aaw") 

- jj.P^iW?" + 2/3|«n«r + ^i2\v\''P) - O}, (3.39) 

it is easy to check that 

6= inf max£'(tu,to)= inf E(u,v). (3.40) 

H3(ij,u)5^(0,0) t>0 («,i>)eA1' 

Note that M C M', one has that B < B. Similarly as (|2.20p . we have S > 0. 

Lemma 3.4. Let /3 > 0, then 

B < min{i?^j , B^^, A}. 

Proof. Step 1. We prove that B < A. Without loss of generality, we may 
assume that G $7. Then there exists p > such that B{0,2p) := {x : |a;| < 
2p} C ri. Let ip £ (7(1(5(0, 2p)) be a nonnegative function with < V' < 1 and 
ip = 1 for I a; I < p. Recall that ([/, V) in Theorem II .61 we define 
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Then it is easy to see that 

|V[/e|2 = / \wu\^ [ \u,f 



2* 



ic/f 



Define 



First we claim the foUowing inequaUties 

Jn JR" 

|(7p*+0(e^), 

Jn 



\UJ 2 2 > 



where C is a positive constant. 

Let < £ <C /). By Proposition 12.11 we have 



\Wi;\^\Ue\^ dx < C e^-^U^{x/e)dx 
n Jp<\x\<2p 

< Ce^ [ U^{x)dx 

J p/e<\x\<2p/e 



< Ce- I \x 

I p/e<\x\<2p/. 



dx = 0(e 



N~2\ 



[ \VU,\^\i;\^ dx < [ |V[/,p= / \WU\^ 
Jn JR" JR« 



and 



< c 



\VU,\\U,\dx 





p<\x\<2p 








p<\x\<2p 


= Ce 


j 1 




J p/e<\x\<2p/e 


< Ce 


j 1 




J p/e<\x\<2p/e 



\VU{x)\\U{x)\dx 
\x\^-^'' dx = 0{e''-^). 



(3.41) 

(3.42) 
(3.43) 
(3.44) 
(3.45) 
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Therefore, 

/ \Vu,fdx= [ \\7U,f\^pf+ [ \\7yjf\U,f + 2 [ il^UeVi^VUe 
Jn JR" Jn Jn 

< f \VU\^dx + 0{e^-^), 
that is, ((02t holds. Note that 

/ {l-i;^')\Uef dx< f e-^\Uix/e)f dx^ f \Uix)f dx 

<C f 1x1-2^ dx = 0(£^), 

J\x\>p/e 

then 

[\u,fdx=[ \Uefdx-f (l-i;''')\Uef dx 
Jn JR" JR« 

> / \Uf dx + 0{e^), 

JR" 

that is, holds. Similarly, holds. Note that 

/ \ue\^dx> f e^-^\U{x/e)\^dx 
Jn J\x\<p 

= e^ I U^dx~e^ [ U^{x)dx 

> Ce^ - Ce^ I \xt'^'' dx = Ce^ + 0(£^-2), 

J\x\>p/e 

that is, p. 451) holds. Similarly, we have 
Jn -/R" 

/ \vef > [ |^P*+0(e^), (3.47) 
Jn JR" 

/ > C£2 + 0(e^-2). (3.48) 
Jn 

Recall that I{U, V) = A, we have 

NA= [ |V[/p + |VFp = / niU^' +2I3U^V^ + fi2V'^' . 
Jr" Jr" 

Combining this with (jOSI - ljallS)) and recahing that Ai,A2<0,2p==2*,iV>5, 
we have for any t > that 

E{tu„tv,) - ^t^ J^{\Vu,\^ + Xiul + \Vv,\^ + X2V^^) 
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2p 

1 

< 



< v4 for e > small enough. (3.49) 

Hence, for e > small enough, there holds 

B <inaxE(tUi,,tv^) < A. (3.50) 
t>o 

Step 2. we shall prove that B < B^-^. This proof is similar to Lemma 12.71 
Recall p.ip and p. 381) . we define t{s) := tu^^.su^^^ that is, 

^ ' NB,, + \s\^PNB^, + \s\P S^2p\u^,\p\u,,\p- 

Note that i(0) = 1. RecaU that \ < p — < 2, by direct computations we 
get that 



t'is) _ pJ,,2p\u^,\P\u^jP 



i^_5o|s|P-2s (2p-2)iVB, 



Ml 



that is, 



and so 



^(^) = ' (2p-2)A^i3,, -^l + ^W)' ^^^-^0' 



This implies that 

Therefore, we deduce from ((3371) and = N/2 that 

+ ispfivi?^, + isr^^2/3K,rK,r) 



S < E {t{s)Uf_,^,t{s)su^.2) 
t{s)^P 



N 



NB^, 
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< as |s| > small enough, 



that is, S < -Bp J . By a similar argument, we can prove that B < Bf^^ . This 



Proof of Theorem 11.31 for the case /3 > 0. Assume that /3 > 0. Since the 
functional E has a mountain pass structure, by the mountain pass theorem (cf. 
[51I3H]) there exists {(u„,w„)} C H such that 



It is standard to see that is bounded in H, and so we may assume 

that (un, Vn) (u, v) wcakly in H. Set w„ = w„ — m and (t„ = w„ — u and use 
the same symbols as in the proof of Theorem ll.3l for the case /3 < 0, we see that 
E'{u,v) = and (IX^ - dXBl) also hold. Moreover, 

0< E{u,v) < E{u,v) + ^{bi+b2) ^ lim E{un,v,,) ^ B. (3.51) 
Case 1. u = 0,v = 0. 

By p.5ip . we have 6i + 62 > 0. Then we may assume that cr„) ^ (0, 0) 
for n large. RecaU Af' in ^M^, by (|3?28l) - (f3^ . it is easy to check that there 
exists tn > such that {tnU!n,tn(Jn) £ Af' and i„ — 1 as n — >■ 00. Then by 
(P:^ and (P3T|) we have 

B = ^{bi + 62) = lim /(a;„,CT„) = lim /(t„w„, t„cr„) > A' = A, 

a contradiction with Lemma 13.41 Therefore, Case 1 is impossible. 

Case 2. u ^ 0, w = or it = 0, v ^ 0. 

Without loss of generality, we may assume that u ^ 0,v = 0. Then w is a 
nontrivial solution of — Au + Xiu — ^^u, and so S > E{u,0) > B^^, a 

contradiction with Lemma 13.41 Therefore, Case 2 is also impossible. 

Since Cases 1 and 2 are both impossible, we have that u ^ 0, u ^ 0. Since 
E'{u, v) = 0, we have {u,v) & M. By B < B and ([33T|l we have E{u, v) = B = 
B. This means \v\) & M C M' and E{\u\, \v\) ^ B = B. By (|339l) and 
(|3.40p . there exists a Lagrange multiplier 7 e M such that 



completes the proof. 



□ 




lim E{un,Vn) = B, lim i?'(u„, ii„) = 0. 



Since E' 



E'{\ul\v\)-jG'i\ul\v\) = 0. 
i\ul\v\)i\ul\v\)^G{\u\,\v\) = OB.iid 




38 



we get that 7 = and so E'{\u\, \v\) = 0. This means that {\u\, \v\) is a least 
energy solution of By the maximum principle, we see that |w|,|w| > Oinfi. 

Therefore, is a positive least energy solution of (|l.ip . This completes 

the proof. □ 



4 Proof of Theorems 11.21 and 11.7 



In this section, we assume that — Ai(51) < Ai = A2 = A < and (3 > {p — 
1) max{/xi, (12}- Define g ■ [{p — 1) max{/xi, (12}, +00) by 



(4.1) 



Then 
9 



1 

'(/?) = ^fi'^"-' -{P- 1)^1^*2) > 0, V /3 > (p - 1) max{/zi, /i2}. 



r 



By direct computations, we have 

l)max{^i,^2}) <p(p- l)p"^max|^J^^, ^2 
Therefore, there exists a unique /3o > {p ^ 1) max{/ii, /i2} such that 
g{M = p{p - 1)^^^ max l^i?^^, /i2^^} > and 
ff(/3) >p(p-l)i-imax{A*?/^ A^2^"}, V /3 > /3o. 



(4.2) 
(4.3) 



Moreover, 



^0 = (p - l)niax{^i,^2}, ifAii=A'2- (4.4) 

Lemma 4.1. Assume that j3 > I3q, where /3o is defined in ^.2^ . Let {ko,lo) be 
in Lemma \2. 1\ Then 

•-iik^~^ < 1, PlJ.2ll~^ < 1- 
I we have 

p-1 



PtJ-iK 

Proof. Let fci = (p/ii)p^, then by 



_pP{ppLl)'^ 



h := hi{ki 

By (|4.3p and direct computations, we get that 
a2ik,,h)^f,2ir'+Pkl/X^'-'-l 



1 

h 
1 

h 



^J,2ll + ki{l - fiik{ ^) 



- 1 



M2^^ 



p-1 
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p-1 



p-i 



-\ 2 



Pl3{pfii)^y^ 



p-1 



(pfll) p- 



- 1 



(p-i)i-2/fp-V7'/^(/3)-i>o. 



Combining this with Lemma 12.31 we have fci > fco, that is, p/iifcp ^ < 1. Simi- 
larly, let I2 — {p^2)~^ 1 then 



a^{h2{h)M) = ip-iy-'/Pp-'fif^^'gW) - 1 > 0. 
By Lemma l2. 31 again, we have h > hi and so ^ < 1- 



□ 



Lemma 4.2. Assume that (3 > /3o, where f3o is defined in Recall ai,a2 

defined in \2. l\) - ^27^ . and {ko,lo) obtained in Lemma lKTi Then 



F{ko,lo) :=det 



dkai{ko,lo) diai{ko,lo) 



< 0. 



9^02(^0,^0) dia2iko,lo) 
Proof. By ai(fco, '0) = c(2{ko, Iq) = we have 

/3fcg^ Iq^ = kg — /ii/cp , i31q^ kf/ = ^0 ^ M2^o 



Then 



= |Mifcr'-(i-p/2)fco"'; 

aia2(A:o, lo) = {p - iWfr' + (p/2 - l)/3Zg/'"'fc?/' 

= |M2;r'-(i-p/2Vo'; 

diai{ko,lo) ^dka2{ko,lo) = |/3fco^^"^^o^^"^ 

= f\/(fco-'-Mifc?-')ao'-/^20- 



Therefore, 



^^(fco, ?o) = [fA^ifcg"' - (1 - p/2)fco"^J [f M2;r' - (1 - P/2)^o ^ 
4 



from Lemma l4.1l 



□ 
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Lemma 4.3. Fix any /ii, /i2 > and /3 > (3o- Let (wq, vq) be a least energy solu- 
tion of with (/ii,/i2,/3) which exists by Theorem ] 1.3[ Recall (^/kou , ^/Iquj) 
in Theorem ] Then 



uofPdx = kP / uj^Pdx. (4.5) 
SI Jn 



Proof. Fix any ^1,^2 > and /3 > ^o- We remark from (|4.ip - (|4.2l) that 
/3o(mi,M2) := /3o is completely determined by /xi,/Z2. Then there exists < e < 
/xi such that for any fi e (/^i — e, /ii + e), we also have /? > l3o{p, ^2)- Then by 
Lemmas 12.11 14.21 and the implicit function theorem, when /ii is replaced by 
functions fco(/i) and are well defined and class for /i S (/ii — ei, /xi +£1) 
for some < ei < e. Recall the definition of i?,A^ and i?, they all depend 
on /i, and we use notations E^,M.^,B{ijl) in this proof, when /xi is replaced by 
/X. Then B{^) = {ko{fJ.) + lo{fj,))Bi g C^((^i — ei,/xi + ei),IR). In particular, 
S'(/xi) := -^Bipi) exists. Note that B = ;B by the proof of Theorem O for 
the case /3 > 0. Then by p. 401) we have 

B(n) = inf max£'u(tu, to). 

ff9(ti,i;)#(0,0) t>0 

Denote 

C = / (|V7XoP + Ai7i2 + |Vi;oP + A2«g), 
Jo 

f {2f3\uonvo\P + fi2\vo\^n, G = I Wo]''^ dx. 
Jn Jn 

There exists <(/i) > such that 

ma.xEfj,{tuo,tvo) = Efj,(t{fi)uo,t{n)va^ , 

where i(/x) > satisfies f{ji,t{ji)) = 0, and 

f{pi,t) ■.= t^P-\fiG + D)-C. 

Note that /(/XI, 1) = 0, f /(/xi, 1) = (2p- 2)(/xiG + Z?) > 0, and /(a*, t(/i)) = 0. 
By the implicit function theorem, there exists < £2 < £1, such that <(/i) G 
C°°((Aii - £2, Ml +£2),^)- By /(/x,i(^)) EEE we see that 

i'(/^i) ^ 



(2p-2)(/xiG + i?) 

By Taylor expansion, we see that t(^) = 1 + t'(/xi)(/x — ^1) + 0{{^ — Mi)^), and 
so 

t^ip) = 1 + 2t'{fi,){p - ^1) + 0((Ai - Ml)')- 
Note that C = mG + D = NB{^ii). Then by (|337l) that 

S(m) < E^{t{^x)uQMp)v(>) = ^t'(M)C^ = i'(M)i?(Mi) 
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= i?(Mi)-^(M-Mi) + 0((M-Min, 
2p 

It follows that ~ -^(a^i) > as ^ /xi, and so B'{^ii) > 

H- Hi 2p 
— Similarly, we have 

G If 

that is, B'(Aii) < -f-. Hence, S'(^i) = -— = -— / \uo\'^p dx. By Theorem 
^ 2p 2p Jj-j 

II. 1[ {^/kouj , ^/Iquj) is also a positive least energy solution of p.ip . Therefore, 

B'(^i) = 2. / uj^Pdx, that is, dill) holds. □ 

2p Jo 

Proof of Theorem ll.2i Let (u, w) be any a positive least energy solution of 
(jl.ip . By Lemma l473l we have 

\u\^Pdx^kP I uj^Pdx. 



By a similar proof of Lemma 14. 3[ that is, by computing i?'(/i2) and B'{f3) 
respectively, we can show that 

\v\^Pdx = lP [ uj^Pdx, and / \u\p\v\p dx ^ k^^^Fo^^ f lo^p dx. 
In Jn Jn Jn 

Therefore, 

\u\P\v\Pdx = ll^^ko''^^ f \u\^Pdx, f lulPlvlPdx^lo^^^k^^^ f \v\^Pdx. 



(4.6) 

Define {u,v) := {^^"^^ ^■^ OiiikoJo) = "2(^0,^0) = and (14. 6p we get 

[ |Vwp + Aw^dx = /" lupPdx, /" \\7v\'^ + Xi'^dx= [ \v\'^Pdx. (4.7) 
Jn Jn Jn 

Then by (fL8| we have 

— / \Wu\^ + Xu^ dx > Bi, — I iVwP + Av^da; > Bi, 



and so 



B = (fco + ^0)^1 = ^ / (|Vu|2 + Aiu2 + |Vv|2 + A2«2) 



42 



> {ko + lo)Bi. 

This implies that 



Combining this with ()4.7p . we see from [9] that u and v are both postive least 
energy solutions of p.6p . Then we see from (MjIj) satisfies (jl.ip that 

that is, uP~^vP = v?^~^ and so u = w. Denote U — u, then {u, v) = (-/fcoC/, VhiU), 
where J7 is a positive least energy solution of ()1.6p . 

Now we assume that SI is a ball in R^, then the positive least energy solution 
of the Brezis-Nirenberg problem (|1.6p is unique (cf. [6]). Therefore, the positive 
least energy solution of is unique. □ 

Proof of Theorem II. 7i The proof is the same as that of Theorem 11.21 □ 

5 Proof of Theorem 11.41 and Theorem 11.51 



This section is devoted to the proof of Theorems 11.41 and 11.51 Recall the defini- 
tion of E,A4 and B, they both depend on f3, and we use notations Ep^J^p^Bp 
in this section. Define B{xo,R) := {x G : \x — xo\ < R}. Consider the 
problem 

j -Au + \2U = H2U^'~^ inB(0,i?), 
\u>0 in B{0, R), u = on 5B(0, R), 



(5.1) 



and the corresponding functional is : Hq{B{0, R)) — > K given by 

Jr{u) = ]-( i\Vu\^ + X2U^)dx-^^,2 f \ufdx. (5.2) 

^ J B(Q,R) ^ Jb{Q,R) 

We need the following energy estimates from the authors' paper |16| . 

Theorem 5.1. (see fW/) Let N >5. Then there exists Rq > and Ci, C2 > 0, 
such that for any < R < Rq, there exists a least energy solution Un of \5.1\) 
and 

1 "-'^ ^r/^ 2iV-4 , , 1 2JV-4 

^ S^/^ ~ C,R^ < MUn) < -M2 ' S""/^ - C2R^. (5.3) 

With the help of Theorem l5.11 we have the following lemma, which improves 
Lemma [01 in case > 6. 
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Lemma 5.1. Let N > 6. Then 



( 1 .^-2 1 N-2 ^ 

supB^ < min + -/i^ ^ 5^/^, + ^ ^^/^ j . 

Proof. Let A'' > 6. For any R > small, we take xji ^ with dist(a;j^, 5r2) = 
3R. Then 

Mx)\<CR, xeB{xR.,3R). (5.4) 

Let V G C^(B(0,2)) with < ■(/' < 1 and ■(/' = 1 in 5(0,1). Define (Pr{x) := 
l-VX^Tp), then 



if xeB{xR,R), 

1 if X eM.^\B{xR,2R), 



__:^;v/!;. \^m{^)\<c/R. (5.5) 



Define it/j := ipRUfj_-^, then by (|5.4p and ()5.5p . it is easy to prove that 

/ |Vufl|^dx< / \\7Uf,,f dx + CR^; 
Jfi Jn 

f \uR\^dx> f \u^,fdx-CR^+^; 
Jn Jn 

[ \uRf'dx> [ \u^,,f dx-CR^+^' . 
Jn Jn 

Therefore, there exists tR > independent of /3 < such that 



mayiEp{tuR,0) = Ep{tRUR,0) 



1 //odV^iflJ' + Aiule)' 



N/2 



~ ^ V (^^Mi - ci?^+2*)^/^* y 

< + Ci?^ for i? > small enough. 

Recall Ur in Theorem 15. 1[ we have Ur{- — xr) ■ ur = 0, and so {tRUR, Ur{- — 
xr)) e for all (3 < 0. Since > 6, one has N > Then we see from 

Theorem O that 



sup < Ep{tRUR,UR{- - xr)) = Ep{tRUR,0) +Efs{0,UR{- - xr)) 

I3<0 

= Efi{tRUR,Q) + jR{UR) 

AT 1 jvr/T 2JV-4 

< B^,, + — Ai^~5'^/2 for i? > smaU enough. 
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1\ —A 

By a similar argument, we also have sup^^g < + wf-^i ^ S^'"^. □ 

Proof of Theorem 11.41 This proof is similar to the proof of [17, Theorem 
1.4] in case = 4. The novelty here is that, with the help of Lemma l5.ll 
we can exclude conclusions (l)-(2) in case iV > 6. Let /3„ < 0, n G N satisfy 
Pn — >■ — OO as n — )■ OO, and (un,Vn) be the positive least energy solutions of (jl.ip 
with 13 = By Lemma 13.11 Ep^^{un,Vn) < A and so is uniformly 

bounded in H by (jl.5p . Passing to a subsequence, we may assume that 

Mn ^ Woo, Vn ^ Woo wcakly in i?o(ri), 

Then, by following the proof of [171 Theorem 1.4] in case = 4, we can prove 
that /BnUl^yP — )■ as n — >■ oo, and passing to a subsequence, one of the 
following conclusions holds. 

(1) Un — > Uoo strongly in HQ{n) and w„ ^ weakly in HQ{n) (so f „ — > for 
almost every x € fl), where Uoo is a positive least energy solution of 

-Au + Aim = fj.i\uf'^^u, u £ HI{9). 

Moreover, 

hm Bp^ = B^, + (5.6) 

n— >oo iV 

(2) Vn — >■ Voo strongly in Hl{Vl) and u„ ^ weakly in i?o(f^) (so u„ for 
almost every x S $7), where Wqo is a positive least energy solution of 

-Aw + X2V = ^al^r^^w, V e -ffo(rJ). 

Moreover, 

hm Bp,^ - B^, + (5.7) 

>oo iV 

(3) («„,«„) (uoo,Woo) Strongly in i?o(^) x -f^o(^) and Uoo • Woo =0 for 
almost a; G f2, where Mqo ^ 0, Voo ^ satisfy 

/ (|VmooP + AimL)= / /.iM^, (5.8) 
i\yv^\^ + \2vl) = I fl2v'^, (5.9) 



lim Bj3^ = i?(uoo, Woo). (5.10) 

n— foo 

Moreover, if Uoo and tJoo are both continuous (we will prove this later), 
then Uoo • Woo = 0, Uoo G C{Q) is a positive least energy solution of 

-Am + Aiu = u e i?o({woo > 0}), 

and Woo G C{Q) is a positive least energy solution of 

-Av + A2W = fl2\vf^^V, V e i?o({Woo > 0}). 

Furthermore, both {woo > 0} and {uoo > 0} are connected domains, and 
{Woo > 0} = n\{uoo > 0}. 
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Note that (|g^ - (|E7)) imply that one of (1) and (2) in Theorem [Ol does not 
hold in some cases. For example, if we assume that — Ai(rj) < Ai < A2 < and 

Ail = ^l2 in TheoremO then B^, + S^/'^ < B^^ + ^^-^^S^/"^, and 

so (2) in Theorem II .41 does not hold, since ()5.7p contradicts with Lemma [3. II 

In particular, Lemma [01 implies that neither (1) nor (2) hold in case > 6. 
That is, only (3) holds if > 6. Therefore, if we assume here the continuity of 
itoo and Woo in conclusion (3), the proof is complete. □ 

From the previous proof, it suffices to prove that Uao and v^o are continuous. 
As pointed out in Remark 11.41 the following proof is completely different from 
that in [TT for the case iV = 4. 

Lemma 5.2. Let (mqo, Vao) be in conclusion (3). Then Uoo — Woo is a least energy 
sign-changing solution of il.l2\) . and Uoo,Woo cife both continuous. 

Proof. Consider the problem (|1.12|) . Its related functional is 

It is standard to prove that P is and its critical points are solutions of (|1.12p . 
Define 

Mu)^ [ i\yu\^ + W-fi,\uf), i = l,2, 
Jn 

S={ue H^{n) : u± ^ 0, Ji(u+) = 0, J2{u-) = O}, 

m = inf P(u). 

ues 

Then any sign-changing solutions of (jl.l2p belong to S. By (|5.8p - (j5.9p . we have 
e S and so m < P(uoo — Woo) = E{uoo,Voo)- For any u G S, we have 
€ Mp for aU 13. Then by (|5.10p we see that 

Piuoo - Voo) = E{uoo,Vao) ^ lini < , ) = P(u) , Vue5, 
and so P{uoo — Woo) < That is, 

P{uoc - Voo) = m. (5-11) 

Step 1. We show that P' {uao — Woo) = 0, and so Uoo — Vca is a least energy 
sign-changing solution of (|1.12l) . 

Thanks to (|5.1ip . the following argument is standard (see [571 [22] for exam- 
ple), and we give the details here for completeness. 

Assume that Uoo— foo is not a critical point of P, then there exists 4> G C^ip) 
such that P'(uoo — Woo)'/' < —1- Then there exists < £0 < 1/10, such that for 
|i — 1| < £0, |s — 1| < £0, l^l < Eo, there holds 

P'{tuoo - svoo + (J(j))4' < -7;- 
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Consider a function < r/ < 1 defined for {t, s) S T = [5, |] x [|, |], such that 

V{t,s) = l, for|i-l|<f, |s-l|<f, 
ri{t,s) = 0, for |t — 1| > eo or |s — 1| > £o- 

Then for |t — 1| < eq, |s — 1| < eoi we have 

=P{tu^-sv^)+ / P'{tu^ - sv^ + 9£ori{t,s)(j))[eor]{t,s)(l)]d9 
Jo 

<P{tUoo - svao) - -eor]{t,s). 
Note that 

sup P{tU^ - SV^) = P(U(x, - Woo) = m, 

t,s>0 

and for |t - 1| > ^ or |s - 1| > ^, there exists <S < ^ such that 

P{tUoo - svoo) <m-S. 
We have, for \t - 1\ < f , |s - 1| < f , that 

P{tUoo - SVoo + sovity < m - y ; 
for f<\t-l\< £0, |s - 1| < £0 or f < |s - 1| < £0, \t - 1\ < Sq, 
P{tUoo - svoo + sovit, s)(p) < P{tUoo - svoo) < m - S; 
for |t — 1| > £0 or |s — 1| > £0, 

P{tUoo - SVoo + SoVit, S)(j)) = P{tUoo - SVoo) < m - S. 

So, 

sup P{tu^ — sv^ + eor]{t,s)(f)) < m — 6. (5-12) 

(t„s)eT 

On the other hand, for £ G [0, £0] , let : T ^ Hq {ft) by {t, s) = tu^o — sv^o + 
£r/(i, s)(j), and H^\T ^'M?hy 

HS.s) = iMhS,s)+),J2ihe{t,s)-)). 

Note that for any {t,s) G dT, we have ri{t,s) = and so he{t,s) = hQ{t,s) = 
tUoo — sVoo- Moreover, 

Ho{t,s) = {Ji{tUoo), J2{sVoo))- 

Then it is easy to see that 

degiH,^{t,s),T, {0,0)) = deg{Ho{t,s),T, (0,0)) = 1, 
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that is, there exists (to, so) S T such that h^g{tQ, sq) E S, which is a contradic- 
tion with (|5TT2)) . 

Step 2. We show that Mqo and Uoo are continuous. 

By Step 1, Moo — Woo is a nontrivial solution of (|1.12p . Then by a Brezis-Kato 
argument (see [S]), we see that Uao — v^o G L'(ri), Vg > 2. In particular, 

Afiu^-i _ ^2w^-i - AiUoo + A2V00 G -L«(f7), \/q> N. 

Then by elliptic regularity theory, Uoo — Woo G iy^'*(ri) with q > N. By 
Sobolev embedding, we have Moo — ^'oo G C{fl). Since Moo = (uoo — Voo)^ 
and Woo = (uoo — ''^oo)", we see that Moo and Woo are both continuous. This 
completes the proof and so completes the proof of Theorem 11.41 □ 

Proof of Theorem II. 5i Let N > 6. Actually, by Theorem 11.41 and Lemma 
15.21 we have proved that the problem (|1.12p has a least energy sign-changing 
solution Moo— Woo- Obviously, Theorem ll.Sl is a direct corollary by letting Ai = A2 
and fJLi = IJL2- n 
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